Repository navigation
Add benchmarks that run on the CI - #164
Open
Vipul-Cariappa wants to merge 3 commits into
Open
Vipul-Cariappa wants to merge 3 commits into
Vipul-Cariappa wants to merge 3 commits into
Conversation
Vipul-Cariappa
force-pushed
the
bench/function-calls
branch
from
October 21, 2026 08:01
f581a86 to
68803bd
Compare
Collaborator
Author
|
If you look at the Benchmark CI run's summary (https://github.com/compiler-research/cppjit/actions/runs/39566226517?pr=164), it provides a report on the benchmarks comparing against Benchmark Report✅ No benchmark runs more than 5% more instructions than @@ Benchmark Diff (geomean) @@
## main #164 +/- ##
============================================================
first/callback 837.2 M 839.3 M +0.25%
first/class 9.275 M 9.37 M +1.02%
first/overload 9.272 M 9.388 M +1.24%
first/primitive 8.547 M 8.638 M +1.07%
first/string 22.06 M 22.22 M +0.75%
first/template 11.17 M 11.26 M +0.80%
first/template_overload 13.59 M 13.72 M +0.97%
hot/callback 9.056 k 9.088 k +0.36%
hot/class 2.869 k 2.87 k +0.06%
hot/overload 2.826 k 2.826 k +0.00%
hot/primitive 2.151 k 2.15 k -0.05%
hot/string 9.877 k 9.874 k -0.03%
hot/template 4.819 k 4.825 k +0.12%
hot/template_overload 2.904 k 2.904 k +0.00%
============================================================
- Slower 0
+ Faster 0
Unchanged 222All benchmarks
|
benchmarks/ measures what calling C++ functions through cppjit costs, as the instructions callgrind counts for a call. Counts do not depend on the machine's load, so one call per case is enough, and runs on different machines with the same toolchain compare. The C++ side (benchmarks/cpp/, built by benchmarks/Makefile into callsDict.so) declares the benchmarked API once and includes it into three namespaces: `hot` for successive calls, and `warmup` and `cold` for first calls, since a function has only one first call. The API covers single-overload functions over every primitive type (by value, pointer, reference and const reference), strings, classes and enums; a non-templated overload set; single and overloaded templates; and overloaded higher order functions taking a Python callback. The bodies only return their input, so the counts are cppjit's overhead. Markers in the library toggle callgrind's counting and dump each count under its test's name. test_calls.py measures every case twice: the first call, from `cold` after warming up on `warmup`, and a successive call, from `hot`. Only the call is counted, with the garbage collector held off; the lookup and the arguments are built before. Every result is checked, so a case that resolves to the wrong overload fails instead of measuring the wrong call. Run without callgrind, the tests only check. callgrind.sh counts the suite. It rebuilds the library with the valgrind headers, and runs Python with ASLR off, a fixed hash seed and a fixed environment, since pointer-keyed hash tables and the environment's size otherwise move the counts from run to run. report.py collects the dumps into JSON, compares two sets of counts into a markdown report that fails above a threshold, and exports counts for github-action-benchmark.
A new Benchmark workflow counts benchmarks/ under callgrind on Linux x86_64, with Release builds against the pinned CppInterOp. On a push to main a separate job, the only one with write access, publishes the counts to gh-pages: github-action-benchmark charts every benchmark (first call and successive calls) over time, and benchmarks/baseline.json keeps the latest counts for pull requests. A pull request builds and counts only itself, then compares against that baseline. Counts depend on the toolchain, so each carries a fingerprint (OS image, glibc, g++, LLVM, Python, valgrind). main is built and counted on the same runner when the baseline is missing, comes from another fingerprint, or shows a regression, which this then confirms. The job fails when a benchmark runs more than 5% more instructions than main. A report job posts the comparison as a PR comment, updated in place like the test results comment. The lint workflow also checks the formatting of the benchmarks' C++ sources.
changelog/2026-10-21-Benchmarks.md describes the benchmarks and their CI: what they measure, how to run them, how pull requests are checked against main, and their known limitations.
Vipul-Cariappa
force-pushed
the
bench/function-calls
branch
from
October 21, 2026 09:23
68803bd to
f5ab456
Compare
Vipul-Cariappa
marked this pull request as ready for review
October 21, 2026 14:21
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Look at the changelog file for a detailed explanation.
We run the benchmarks under callgrind and count instructions. We publish the results on GitHub Pages. A PR is compared against
main's benchmark results and is considered a failure if it is more than 5% slower. We may want to adjust things depending on what we see in our CI later. We may want to extend this with cache misses later.The current benchmarks are focused on templated and non-templated function calls with and without multiple overloads.
Once merged, we will need to turn on GitHub Pages on this repo.